Project 4 : Vehicle Silhouette Classfication and PCA¶
Submitted by Aman Kumar Agarwal¶
Question :¶
The data contains features extracted from the silhouette of vehicles in different angles. Four "Corgie" model vehicles were used for the experiment: a double decker bus, Cheverolet van, Saab 9000 and an Opel Manta 400 cars. This particular combination of vehicles was chosen with the expectation that the bus, van and either one of the cars would be readily distinguishable, but it would be more difficult to distinguish between the cars.
Attribute Information¶
Input variables:¶
COMPACTNESS (average perim)**2/area
CIRCULARITY (average radius)**2/area
DISTANCE CIRCULARITY area/(av.distance from border)**2
RADIUS RATIO (max.rad-min.rad)/av.radius
PR.AXIS ASPECT RATIO (minor axis)/(major axis)
MAX.LENGTH ASPECT RATIO (length perp. max length)/(max length)
SCATTER RATIO (inertia about minor axis)/(inertia about major axis)
ELONGATEDNESS area/(shrink width)**2
PR.AXIS RECTANGULARITY area/(pr.axis length*pr.axis width)
MAX.LENGTH RECTANGULARITY area/(max.length*length perp. to this)
SCALED VARIANCE (2nd order moment about minor axis)/area ALONG MAJOR AXIS
SCALED VARIANCE (2nd order moment about major axis)/area ALONG MINOR AXIS
SCALED RADIUS OF GYRATION (mavar+mivar)/area
SCALED RADIUS OF GYRATION (mavar+mivar)/area
SKEWNESS ABOUT (3rd order moment about major axis)/sigma_min**3 MAJOR AXIS
SKEWNESS ABOUT (3rd order moment about minor axis)/sigma_maj**3 MINOR AXIS
SKEWNESS ABOUT (3rd order moment about minor axis)/sigma_maj**2 MINOR AXIS
HOLLOWS RATIO (area of hollows)/(area of bounding polygon)
CLASS : Different classes of vehicles.
Part 1 : Data Preprocessing ¶
Importing Libraries¶
In [1]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
sns.set(color_codes=True)
import warnings
warnings.filterwarnings("ignore", category=FutureWarning)
warnings.filterwarnings("ignore", category=RuntimeWarning)
Loading the dataset¶
In [2]:
dataset = pd.read_csv("D:/Study/AI-ML/Dataset/vehicle.csv")
Overview of the data¶
In [3]:
dataset.head()
Out[3]:
In [4]:
dataset.info() # Total of 846 entries but individual attributes total non-null is not 846. Indicates missing values.
In [5]:
dataset.isnull().sum() # Many attributes contain missing values
Out[5]:
In [6]:
dataset.describe().T
Out[6]:
Transforming data-types¶
In [7]:
dataset['class'] = dataset['class'].astype('category')
In [8]:
data = dataset.drop("class",axis=1)
Data Cleaning and Exploratory Data Analysis (EDA)¶
In [9]:
dataset.plot(kind="box",figsize=(20,10))
# we see that few columns have outliers while few columns do not have outliers.
# Attributes with missing values and outliers are replaced by median
# Attributes with missing values and no outliers are replaced by mean
# Outliers are replaced by the median
Out[9]:
In [10]:
# Attributes 4,5,6,11,12,14,15,16 have outliers. Treat missing values with median
# Attributes 1,2,3,7,8,9,10,13,17,18 do not have outliers. Treat missing values with mean.
outlier_attributes = [3,4,5,10,11,13,14,15]
non_outlier_attributes = [0,1,2,6,7,8,9,12,16,17]
In [11]:
for i in dataset.columns[outlier_attributes]:
median = dataset[i].median()
dataset[i] = dataset[i].fillna(median)
In [12]:
for i in dataset.columns[non_outlier_attributes]:
mean = dataset[i].mean()
dataset[i] = dataset[i].fillna(mean)
In [13]:
dataset.isnull().sum()
Out[13]:
In [14]:
dataset.columns[:18]
Out[14]:
In [15]:
# Handling outliers
for col in dataset.columns[:18]:
q1 = dataset[col].quantile(0.25)
q3 = dataset[col].quantile(0.75)
iqr = q3 - q1
low = q1 - (1.5*iqr)
high = q3 + (1.5*iqr)
dataset.loc[(dataset[col]<low) | (dataset[col] > high),col] = dataset[col].median()
In [16]:
dataset.plot(kind="box",figsize=(20,10))
# Outliers have been removed
Out[16]:
1. Univariate Analysis.¶
In [17]:
dataset.hist(figsize=(15,15))
plt.show()
# Skewness exists among the different attributes.
In [18]:
sns.distplot(dataset['compactness'],hist=False) #Almost normal with a little skew
Out[18]:
In [19]:
fig,axes = plt.subplots(nrows=3,ncols=3,figsize=(15,7))
sns.distplot(dataset['circularity'],hist=False,ax = axes[0][0])
sns.distplot(dataset['distance_circularity'],hist=False,ax = axes[0][1])
sns.distplot(dataset['scatter_ratio'],hist=False,ax = axes[0][2])
sns.distplot(dataset['pr.axis_rectangularity'],hist=False,ax = axes[1][0])
sns.distplot(dataset['scaled_variance'],hist=False,ax = axes[1][1])
sns.distplot(dataset['scaled_radius_of_gyration'],hist=False,ax = axes[1][2])
sns.distplot(dataset['skewness_about'],hist=False,ax = axes[2][0])
sns.distplot(dataset['skewness_about.1'],hist=False,ax = axes[2][1])
sns.distplot(dataset['hollows_ratio'],hist=False,ax = axes[2][2])
# Skewness axists between the attributes.
Out[19]:
2. Bivariate and Multivariate Analysis.¶
In [20]:
sns.scatterplot(dataset['compactness'],dataset['circularity'],hue=dataset['class'])
# The compactness in the vans range from 82-100 whereas the compactness in the
# other vehicle types have a greater spread.
Out[20]:
In [21]:
sns.scatterplot(dataset['compactness'],dataset['distance_circularity'],hue=dataset['class'])
# Buses have a maximum distance circularity of 90 whereas it is greater for other classes of vehicles.
Out[21]:
In [22]:
sns.scatterplot(dataset['scatter_ratio'],dataset['circularity'],hue=dataset['class'])
# Moderately strong linear co-relation
Out[22]:
In [23]:
sns.scatterplot(dataset['scatter_ratio'],dataset['distance_circularity'],hue=dataset['class'])
# Strong positive linear corelation
Out[23]:
In [24]:
fig,axes = plt.subplots(nrows=2,ncols=2,figsize=(15,10))
sns.scatterplot(dataset['circularity'],dataset['elongatedness'],hue=dataset['class'],ax=axes[0][0])
sns.scatterplot(dataset['distance_circularity'],dataset['elongatedness'],hue=dataset['class'],ax=axes[0][1])
sns.scatterplot(dataset['radius_ratio'],dataset['elongatedness'],hue=dataset['class'],ax=axes[1][0])
sns.scatterplot(dataset['scatter_ratio'],dataset['elongatedness'],hue=dataset['class'],ax=axes[1][1])
plt.show()
# Very strog negative co-relation between elongatedness and other attributes such as circularity,scatter_ratio etc.
In [25]:
fig,axes = plt.subplots(nrows=2,ncols=2,figsize=(15,10))
sns.scatterplot(dataset['circularity'],dataset['pr.axis_rectangularity'],hue=dataset['class'],ax=axes[0][0])
sns.scatterplot(dataset['distance_circularity'],dataset['pr.axis_rectangularity'],hue=dataset['class'],ax=axes[0][1])
sns.scatterplot(dataset['scatter_ratio'],dataset['pr.axis_rectangularity'],hue=dataset['class'],ax=axes[1][0])
sns.scatterplot(dataset['elongatedness'],dataset['pr.axis_rectangularity'],hue=dataset['class'],ax=axes[1][1])
plt.show()
# Strong positive co-relation between axis-rectangularity and other attributes such as circularity,scatter_ratio etc.
# Strong negative linear co-relation between axis-rectangularity and elongatedness.
In [26]:
fig,axes = plt.subplots(nrows=1,ncols=2,figsize=(15,5))
sns.scatterplot(dataset['scatter_ratio'],dataset['max.length_rectangularity'],hue=dataset['class'],ax=axes[0])
sns.scatterplot(dataset['elongatedness'],dataset['max.length_rectangularity'],hue=dataset['class'],ax=axes[1])
# Vans have least scatter ratio with respect to length rectangularity followed by cars and buses.
# On the other hand, vans have highest elongatedness with respect to length rectangularity followed by cars and buses.
Out[26]:
In [27]:
fig,axes = plt.subplots(nrows=3,ncols=2,figsize=(15,15))
sns.scatterplot(dataset['compactness'],dataset['scaled_variance'],hue=dataset['class'],ax=axes[0][0])
sns.scatterplot(dataset['circularity'],dataset['scaled_variance'],hue=dataset['class'],ax=axes[0][1])
sns.scatterplot(dataset['distance_circularity'],dataset['scaled_variance'],hue=dataset['class'],ax=axes[1][0])
sns.scatterplot(dataset['radius_ratio'],dataset['scaled_variance'],hue=dataset['class'],ax=axes[1][1])
sns.scatterplot(dataset['scatter_ratio'],dataset['scaled_variance'],hue=dataset['class'],ax=axes[2][0])
sns.scatterplot(dataset['elongatedness'],dataset['scaled_variance'],hue=dataset['class'],ax=axes[2][1])
plt.show()
# Strong positive co-relation between scaled_variance and other attributes such as circularity,scatter_ratio etc.
# Strong negative linear co-relation between scaled_variance and elongatedness.
In [28]:
sns.pairplot(dataset,diag_kind="kde")
# We see that the attributes are highly linearly related to each other. Either positively or negatively.
Out[28]:
In [29]:
sns.pairplot(dataset,diag_kind="kde",hue="class")
# Attributs have a strong linear co-relation with most of the other attributes
# but tend to have a storng linear corelation with elongatedness.
Out[29]:
In [30]:
corr = dataset.corr()
plt.figure(figsize=(13,7))
sns.heatmap(corr,annot=True)
#Many attributes have strong negative and strong positive co-relation
Out[30]:
Part 2 : Splitting Data & Model Building
Seperating independent and Dependent attributes¶
In [31]:
# Splitting data on independent and target variables
X = dataset.iloc[:,0:18]
y = dataset.iloc[:,-1]
Building model without Principal Component Analysis (PCA)¶
In [32]:
from sklearn.model_selection import train_test_split # Library for splitting the dataset
from sklearn.metrics import confusion_matrix # Library for Evaluation of result
from sklearn.metrics import classification_report # Library for vewing the classification report
from sklearn.naive_bayes import GaussianNB
from sklearn import svm
from scipy.stats import zscore
In [33]:
# Normalizing data
X_z = X.apply(zscore)
In [34]:
X_train, X_test, y_train, y_test = train_test_split(X_z,y,test_size = 0.3, random_state = 10)
-----------------------------------------Support Vector Machine--------------------------------------------¶
Creating the classifier and Training the model.¶
In [35]:
classifier_SVM_rbf_kernel = svm.SVC(gamma='auto')
In [36]:
classifier_SVM_rbf_kernel.fit(X_train,y_train)
Out[36]:
Predicting the output of test data and measuring accuracy score¶
In [37]:
y_pred_SVM_rbf = classifier_SVM_rbf_kernel.predict(X_test)
In [38]:
svm_score_rbf = classifier_SVM_rbf_kernel.score(X_test,y_test)
svm_score_rbf
Out[38]:
Confusion matrix for overall score¶
In [39]:
svm_cm = confusion_matrix(y_test,y_pred_SVM_rbf)
svm_cm
Out[39]:
Classification Report for class-level metrics such as precision and Recall¶
In [40]:
svm_cr = classification_report(y_test,y_pred_SVM_rbf)
print(svm_cr)
Exploring Other Kernels in SVM¶
In [41]:
classifier_SVM_linear_kernel = svm.SVC(kernel = "linear",gamma='auto')
In [42]:
classifier_SVM_linear_kernel.fit(X_train,y_train)
Out[42]:
In [43]:
y_pred_SVM_linear = classifier_SVM_linear_kernel.predict(X_test)
In [44]:
svm_score_linear = classifier_SVM_linear_kernel.score(X_test,y_test)
svm_score_linear
Out[44]:
In [45]:
svm_cm2 = confusion_matrix(y_test,y_pred_SVM_linear)
svm_cm2
Out[45]:
In [46]:
svm_cr2 = classification_report(y_test,y_pred_SVM_linear)
print(svm_cr2)
--------------------------------------------------Naive Bayes--------------------------------------------------------¶
Creating the classifier and Training the model.¶
In [47]:
classifier_NB = GaussianNB()
In [48]:
classifier_NB.fit(X_train,y_train)
Out[48]:
Predicting the output of test data and measuring accuracy score¶
In [49]:
y_pred_NB = classifier_NB.predict(X_test)
In [50]:
NB_score = classifier_NB.score(X_test,y_test)
NB_score
Out[50]:
Confusion matrix for overall score¶
In [51]:
NB_cm = confusion_matrix(y_test,y_pred_NB)
NB_cm
Out[51]:
Classification Report for class-level metrics such as precision and Recall¶
In [52]:
NB_cr = classification_report(y_test,y_pred_NB)
print(NB_cr)
# Low accuracy and recall for certain classes. Aims at improvising by applying PCA.
Applying PCA¶
In [53]:
from sklearn.decomposition import PCA
In [54]:
pca = PCA()
In [56]:
X_pca = X_z
In [57]:
pca.fit(X_pca)
cum = np.cumsum(pca.explained_variance_ratio_)
sns.pointplot(x=np.arange(1,19),y=cum)
Out[57]:
In [58]:
np.cumsum(pca.explained_variance_ratio_)
# First 9 attributes are able to define 98% of the information.
# So the dimensions can be halved from 18 to 9 with a loss of only 2% information.
# First 7 dimensions explains more thn 95% variance.
Out[58]:
In [59]:
plt.figure(figsize=(10,6))
sns.barplot(x=np.arange(1,19),y=pca.explained_variance_ratio_,label = "Individual Explained Variance")
plt.step(x = np.arange(1,19),y=cum,where='mid',label = "Cummulative Explained Variance")
plt.ylabel('Explained variance ratio')
plt.xlabel("Principal Components")
plt.legend(loc = 'center right')
plt.show()
In [60]:
pca = PCA(n_components=9)
X_pca = X_z
pca.fit(X_pca)
X_pca = pca.transform(X_pca)
In [61]:
X_pca = pd.DataFrame(X_pca)
Building model with pca¶
In [62]:
X_train_pca, X_test_pca, y_train, y_test = train_test_split(X_pca,y,test_size = 0.3, random_state = 10)
Support Vector Machine¶
Creating the classifier and Training the model.¶
In [63]:
classifier_SVM_rbf_kernel_pca = svm.SVC(gamma='auto')
In [64]:
classifier_SVM_rbf_kernel_pca.fit(X_train_pca,y_train)
Out[64]:
Predicting the output of test data and measuring accuracy score¶
In [65]:
y_pred_SVM_rbf_pca = classifier_SVM_rbf_kernel_pca.predict(X_test_pca)
In [66]:
svm_score_rbf_pca = classifier_SVM_rbf_kernel_pca.score(X_test_pca,y_test)
svm_score_rbf_pca
Out[66]:
Confusion matrix for overall score¶
In [67]:
svm_cm_pca = confusion_matrix(y_test,y_pred_SVM_rbf_pca)
svm_cm_pca
Out[67]:
Classification Report for class-level metrics such as precision and Recall¶
In [68]:
svm_cr_pca = classification_report(y_test,y_pred_SVM_rbf_pca)
print(svm_cr_pca)
Linear Kernel¶
Creating the classifier and Training the model.¶
In [69]:
classifier_SVM_linear_kernel_pca = svm.SVC(kernel = "linear",gamma='auto')
In [70]:
classifier_SVM_linear_kernel_pca.fit(X_train_pca,y_train)
Out[70]:
Predicting the output of test data and measuring accuracy score¶
In [71]:
y_pred_SVM_linear_pca = classifier_SVM_linear_kernel_pca.predict(X_test_pca)
In [72]:
svm_score_linear_pca = classifier_SVM_linear_kernel_pca.score(X_test_pca,y_test)
svm_score_linear_pca
Out[72]:
Confusion matrix for overall score¶
In [73]:
svm_cm_pca2 = confusion_matrix(y_test,y_pred_SVM_linear_pca)
svm_cm_pca2
Out[73]:
Classification Report for class-level metrics such as precision and Recall¶
In [74]:
svm_cr_pca2 = classification_report(y_test,y_pred_SVM_linear_pca)
print(svm_cr_pca2)
Naive Bayes¶
Creating the classifier and Training the model.¶
In [75]:
classifier_NB_pca = GaussianNB()
In [76]:
classifier_NB_pca.fit(X_train_pca,y_train)
Out[76]:
Predicting the output of test data and measuring accuracy score¶
In [77]:
y_pred_NB_pca = classifier_NB_pca.predict(X_test_pca)
In [78]:
NB_score_pca = classifier_NB_pca.score(X_test_pca,y_test)
NB_score_pca
Out[78]:
Confusion matrix for overall score¶
In [79]:
NB_cm_pca = confusion_matrix(y_test,y_pred_NB_pca)
NB_cm_pca
Out[79]:
Classification Report for class-level metrics such as precision and Recall¶
In [80]:
NB_cr_pca = classification_report(y_test,y_pred_NB_pca)
print(NB_cr_pca)
Part 3 : Comparing models with and without PCA ¶
Initializing scores arrays¶
In [81]:
SVM_rbf = []
SVM_linear = []
NB = []
Accuracy scores without PCA¶
In [82]:
SVM_rbf.append(svm_score_rbf)
SVM_linear.append(svm_score_linear)
NB.append(NB_score)
In [83]:
SVM_rbf,SVM_linear,NB
Out[83]:
Comparing various accuracies with different principal components selection¶
In [84]:
# Based on the above plot, we select the first 7 components as they explain more than 95% of the data
# and progressively see how the model's accuracy changes with increase in principal components.
In [85]:
for n in range(7,19):
X_pca = X_z # Resetting the X value for each iteration
pca = PCA(n_components=n) # Initializing pca with n components for each iteration
pca.fit(X_pca)
X_pca = pca.transform(X_pca)
X_pca = pd.DataFrame(X_pca)
#Splitting the transformed X (X_pca) for building the model
X_train_pca, X_test_pca, y_train, y_test = train_test_split(X_pca,y,test_size = 0.3, random_state = 10)
# Model 1 - Support Vector Machine
## RBF Kernel
classifier_SVM_rbf_kernel_pca = svm.SVC(gamma='auto')
classifier_SVM_rbf_kernel_pca.fit(X_train_pca,y_train)
SVM_rbf.append(classifier_SVM_rbf_kernel_pca.score(X_test_pca,y_test))
## Linear Kernel
classifier_SVM_linear_kernel_pca = svm.SVC(kernel = "linear",gamma='auto')
classifier_SVM_linear_kernel_pca.fit(X_train_pca,y_train)
SVM_linear.append(classifier_SVM_linear_kernel_pca.score(X_test_pca,y_test))
## Naive Bayes
classifier_NB_pca = GaussianNB()
classifier_NB_pca.fit(X_train_pca,y_train)
NB.append(classifier_NB_pca.score(X_test_pca,y_test))
In [86]:
labels = [0,7,8,9,10,11,12,13,14,15,16,17,18]
values_rbf = []
values_linear = []
values_NB = []
for i in range(0,len(SVM_rbf)):
values_rbf.append(SVM_rbf[i])
values_linear.append(SVM_linear[i])
values_NB.append(NB[i])
In [87]:
plt.figure(figsize=(15,8))
sns.pointplot(x = labels,y=values_rbf,color="#33FF3C",label="RBF"),
sns.pointplot(x = labels,y=values_linear,color="#4C33FF",label="Linear"),
sns.pointplot(x = labels,y=values_NB,color="#FF4933",label="Naive Bayes")
leg = plt.legend(labels=['RBF','Linear','Naive Bayes'], loc ='center',prop={'size':16})
leg.legendHandles[0].set_color("#33FF3C")
leg.legendHandles[1].set_color("#4C33FF")
leg.legendHandles[2].set_color("#FF4933")
plt.title("Comparing model accuracy without PCA and with PCA\n")
plt.xlabel("Principal Components")
plt.ylabel("Accuracy")
plt.show()
In [88]:
# If we consider just accuracy, then with 14 components we get the maximum accuracy in all the three models.
# But if we want to consider according to the information content captured by the components,
# then with only 7 components, more than 95% variance is covered. This also gives a good increase
# in accuracy in Naive bayes but a decrease in the SVM Models.
# Therefore, we choose 9 dimensions which shows a stable and good increase in accuracy in all the 3 models.
# This allows us to therefore reduce the number of dimensions from 18 to 9 which is a significant good reduction.
Part 4 : Conclusion ¶
Insights from EDA¶
- Data has missing values.
- Data has outliers present.
- There is moderate amount of skewness in the data.
- There exists strong linear co-relation between many attributes.
- The linear co-relations are both negative and postive.
- The compactness in the vans range from 82-100 whereas the compactness in the other vehicle types have a greater spread.
- Buses have a maximum distance circularity of 90 whereas it is greater for other classes of vehicles.
- Very strog negative co-relation between elongatedness and other attributes such as circularity,scatter_ratio etc.
- Strong positive co-relation between axis-rectangularity and other attributes such as circularity,scatter_ratio etc.
- Strong negative linear co-relation between axis-rectangularity and elongatedness.
- Vans have least scatter ratio with respect to length rectangularity followed by cars and buses.
- On the other hand, vans have highest elongatedness with respect to length rectangularity followed by cars and buses.
- Strong positive co-relation between scaled_variance and other attributes such as circularity,scatter_ratio etc.
- Strong negative linear co-relation between scaled_variance and elongatedness.
- The heatmap also shows the existence of large number of strong co-relations between the different attributes.
Model Inferences¶
Without Applying PCA
- The Support vector classifier gave an accuracy of 96% on the rbf kernel.
- The Support vector classifier gave an accuracy of 93.30% on the Linear kernel.
- The Naive Bayes Classifier gave a low accuracy of approximately 60%.
After Applying PCA (7 components)
- The Support vector classifier gave an accuracy of 96% on the rbf kernel.
- The Support vector classifier gave an accuracy of 85.40% on the Linear kernel.
- The Naive Bayes Classifier gave a much higher accuracy of 79.92% which was previously 60%.
- Therefore Principal Component (PCA) Analysis helped in increasing the model performance as well as reducing the dimensions significantly.
Regularizations¶
- The Principal component analysis (PCA) technique was run through multiple values of components to help compare the accuracy and stabality of the models with the increasing number of components.
- With 7 components, the analysis was able to cover or capture more than 95% variance in the data.
- With 9 components, the analysis was able to cover or capture more than 98% variance in the data. Thereby allowing us to reduce the dimensions by half with only 2% information loss.
- The acccuracy of the various models also changed with the increase in the components.
- The models saw an increase in accuracy by 5% from 85.4% to 90% in SVM Linear Kernel, and an increase in accuracy by 8% from 79.92% to 87% in Naive Bayes.
- The SVM Classifier was tested for different kernels such as rbf and linear.
- If we consider just accuracy, then with 14 components we get the maximum accuracy in all the three models.
- But if we want to consider according to the information content captured by the components, then with only 7 components, more than 95% variance is covered. This also gives a good increase in accuracy in Naive bayes but a decrease in the SVM models.
- Therefore, we choose 9 dimensions which shows a stable and good increase in accuracy in all the 3 models.
- PCA therefore allows us to reduce the number of dimensions from 18 to 9 which is a significant good reduction.
Learnings¶
- Principal Component Analysis (PCA) helps in reducing the dimensions in a dataset where there exists linear co-relation between the attributes.
- PCA uses the concept of eigen decomposition to absorb the information content and capture as much variance as possible in the data.
- PCA aims at minimal information loss to help reduce the dimensions in the dataset.
- What was clearly visible as separate gaussians in original dimensions before is not visible any more. This is due to the fact that PCA dimesions are composite of the original dimensions.
- Each dimension contains a part of information from the other dimensions.
- Standardization of data to a common scale is very important in PCA to prevent any biasing.
- PCA must only be used when the data is linearly co-related.
- Outliers can pull down the algorithm's accuracy significantly in PCA and therefore they must be handled while cleaning the data.
- Support Vector Classifier in this given case gives the highest and most stable accuracy of 96%.
- Therefore we can use Support Vector Machine in the classsification of the vehicle silhouettes.
In [ ]: